1. Technical Field
This invention relates in general to improvements in the field of computer systems having backup/restore or archive/retrieve subsystems. More particularly, this invention relates to a method and system for reducing the storage requirements of backup subsystems in client-server environments.
2. Description of the Background Art
In a data processing system, a backup/restore subsystem, usually referred to as backup subsystem, is typically used as a means to save a recent copy or version of a file, plus some number of earlier versions of the same file, on some form of backup storage devices such as magnetic disk drives, tapes, or optical storage devices. The backup subsystem is used as a means of protecting against loss of data in a given data processing system. For example, if an on-line version of a file is destroyed or corrupted because of power failure, hardware, or software error, user error or some other type of problem, the latest version of that file which is stored in a backup subsystem can be restored and therefore the risk of loss of data is minimized. Another important use of backup subsystems is that even if failures do not occur, but files or data are deleted or changed (either accidentally or intentionally), those files or data could be restored to their earlier state thus minimizing the loss of data.
Therefore, it can readily be apparent that backup subsystems are and will remain an important part of the field of data processing.
A closely related concept to the backup subsystem is a method and system called archive/retrieve, usually referred to as an archive subsystem. Archiving refers to making copies of files on lower cost storage such as tape so that files can be deleted from more expensive storage such as disk storage. Since disk storage is frequently being updated, an archival copy also allows the state of a collection of data to be captured for later reference, even if the primary copy of the data is not going to be deleted. An example would be the archiving of a set of financial data at the end of a fiscal period. Although the improved method of carrying out the backup disclosed in this application is primarily described for a backup system, it will be obvious to a person of ordinary skill in the art of data processing that the systems and methods described herein are also applicable to archive systems or other related storage management systems.
At the present time, the majority of backup subsystems run on host systems located in a data processing environment. Typically, a new version (also referred to as changed version) of a file is backed-up based on a predetermined schedule such as, at the end of each day, or after each time that a file has been updated and saved.
Backup systems generally consume large amount of storage media because multiple versions of large amounts of data are being backed up on a regular basis. Therefore, those engaged in the field of data processing and especially in the field of backup/restore systems are continuously striving to find improved methods and systems to reduce the storage demand in backup systems. Current backup systems typically utilize one or both of the following methods to enable the storage of and retrieval of multiple versions of a given file. These are: (1) the full backup method and (2) the incremental backup method.
The full backup method is the most basic method used which requires the backup of an entire collection of files, or a file system, regardless of whether individual files in that collection have been updated or not. Furthermore, in the full backup method multiple full versions of each file are maintained on a storage device. Since maintaining multiple full copies of many files consumes substantial amount of storage, some type of compression technique is sometimes used to reduce the amount of data stored. Compression techniques basically rely on the presence of redundancy within the file, so called intra-file redundancy, in order to achieve this reduction. The most common method is the use of a method of file compression known as Lempel-Ziv method (also known as Adaptive Dictionary Encoder or LZ coding) described in a book by T. C. Bell et. al., titled Text Compression, pp 206-235. The essence of Lempel-Ziv coding is that phrases are replaced with a pointer to where they have occurred earlier in the text, thereby saving the storage space associated with multiple occurrence of any given phrase. This is a general method which can be applied to any file and typically results in compression ratios of the order of between 2 and 3.
Incremental backup method is an alternative to full backup method used in backup systems where only those files, in any given collection of files, are backed up which have been changed since the previous incremental or full backup.
It is apparent to those skilled in the art that in any given backup system, the higher the backup frequency, the more accurately the backup copy will represent the present state of data within a file. Considering the large volume of data maintained and continuously generated in a typical data processing system, the amount of storage, time, and other resources associated with backing up data are very substantial. Thus, those skilled in the art are continuously engaged in searching for better alternatives and more storage and time efficient systems and methods for backing up data.
Aside from the compression technique which is heavily utilized to reduce storage requirement in a backup system, there exists a quite different method of achieving reduction in file size, known as delta versioning. Delta versioning has never been used in any backup system.
Delta versioning which is also referred to as "differencing" or "deltaing" relies on comparison between two files where multiple version of a file is saved in a form of a "base" file, also called a "base version" of a file, together with predetermined number of small files which represent only the changes to the base file. The small files, also referred to as "delta" files or "difference" files, contain the difference or delta from the base file. Delta files are generated as a result of comparing the base file with a later (newly arrived) or an earlier version of the base file. Thus this method of storage reduction exploits redundancy between files, or "inter-file" redundancy, in order to achieve reduction in storage requirement. This method which is used in the software art of Source Code Control Systems, discussed in a reference below, can provide substantial storage savings in backup applications, since frequently the selection of a file for incremental backup occurs after a small change has been made to that file. Therefore, since many copies are frequently made in backup systems to files that differ only slightly from one another, the differencing method offers great potential for substantial reduction in the amount of data stored in backup subsystems. At the present time none of the backup systems utilize delta versioning.
Delta versioning falls into two general classes: one is where the base file is the oldest version of a file and the delta files represent newer versions. This method is referred to as "forward" deltas. The other is where the base file is the latest version of a file and the delta files represent older versions. This method is referred to as "reverse" deltas. The "reverse" delta is the more common method because usually the most utilized version of a file is the last version created.
A technical paper by M. J. Rochkind, titled "The Source Code Control System", IEEE Transaction on Software Engineering, Vol. SE-1, No. 4, Dec. 1975, PP 364-370, teaches a software tool, known as source code control system (SCCS) which is designed to help managing changes to a source code (source program) in the field of software development tools. In SCCS environment, every time a module (file) is changed the change is stored as a discrete delta where the space required to store a delta is only slightly greater than the amount of text inserted by that delta. However, Rochkind does not teach or suggest the use of delta files in a backup and archiving subsystem in either a central or a client-server environment as a means for reducing the storage requirements of such subsystems.
U.S. Pat. No. 4,912,637 issued on Mar. 27, 1990 to C. R. Sheedy et al., teaches a system for preserving, generating, and merging various versions of the same file by a modified delta method. Sheedy teaches using an indexed line file where every line active in any version of a given file is stored, together with a variant history file where the history of the status of each line in various versions is recorded. Using these two files, any desired version of a program may be generated directly without the need for creating any of the intermediate versions. However, Sheedy does not teach or suggest the use of this modified method in backup and archiving systems in either a central processing or a client/server environment as a means for reducing the storage requirement of a packup system.
U.S. Pat. No. 5,263,154, issued on Nov. 16, 1993 to L. E. Eastridge et al., teaches a method and system for incremental backup copying of a file in a data processing system which minimizes the suspension of the data processing system during such backup copying. This is done by first physically backing up a data set on a storage subsystem on a scheduled or opportunistic basis. Thereafter, creating side-files of the data set modified. The side-files are then used in the next scheduled or opportunity to update the backed-up data set. However, Eastridge does not teach or suggest the use of delta files as a means of minimizing storage requirement in a backup and archiving subsystem in either a central processing environment or a client-server environment.
U.S. Pat. No. 5,278,979 issued on Jan. 11, 1994 to R. D. Foster, et al., teaches a method and system in the field of software code development known as Single Entity Versioning where by creating and maintaining a unique version identification and a control data file, multiple versions of source data is efficiently stored in a single entity. However, Foster does not teach or suggest the use of delta versioning or Single Entity Versioning as a means of minimizing storage requirement in a backup and archiving subsystem in either a central processing environment or a client-server environment.
Considering that the amount of the data generated on the daily basis by the computers is growing at a very fast rate, there is a need for an improved and innovative method and system to reduce the storage requirements of backup systems in central data processing systems and further in client-server environments which as will be discussed below present unique backup issues.
Backup Subsystems in a Client-Server Environment
Recently, the emergence of low cost local area networking, personal computer, and workstation technology has promoted a new type of data processing architecture known as the , "client-server" system or environment. A client-server system 10, as shown in FIG. 1, typically consists of: (1) client computers (also referred to as clients) 11 such as personal computers or workstations with their own local storage medium 12 such as disk storage devices; (2) a local area network (also referred to as LAN or network) 13 such as an Ethernet or a Token Ring which links the clients to the LAN server(s); and (3) one or more LAN server computers 14 such as a personal computer or perhaps a workstation with its own local storage mediums 15 such as disk storage devices, tape storage and/or optical storage devices.
In a client-server environment, the majority of the data processing is usually carried out at the clients which are connected by a local area or other network to a LAN server. The LAN server usually contains various programs or data which are commonly used by many of the clients. Computer users which usually use clients to carry on their data processing tasks, are generally in control of the client computers whereas the LAN server(s) is usually administered by an expert administrator of a data processing (computing) center.
The client-server environment presents a number of major issues as relates to data processing, integrity, and backup of such data. One major concern in the client-server environment is that a substantial amount of important data is located on client subsystems which lack the security, reliability or care of administration that is typically applied to the server machine(s). There is a further concern that data may accidentally be lost from a client computers, because the users of such computers do not take time and necessary care to back up the data on a regular basis. There is yet another concern that the amount of data residing on the clients are so substantial that even if a client-server backup subsystem could be developed to attempt to backup all these data, the amount of backup storage required to save all the data on the clients would be inordinate and impractical. The lack of an efficient backup system and method has been a major barrier to the adoption and rapid growth of client-server technology despite its many attractive features.
Recently a number of client-server backup systems have been developed to alleviate some of the concerns listed above. An example is an IBM's ADSM (ADSTAR Distributed Storage Manager) product. This technology overcomes some of the deficiencies mentioned above by making backup copies of the client data on a backup server. The client copies are made automatically without user involvement and are stored on storage devices which are administered by the backup server.
A typical client-server backup subsystem such as ADSM operates as follows. In the client computer a program exists, known as the client backup program, which at pre-specified or periodic times is activated and makes contact with a program residing on the backup server, known as the server backup program. After establishing contact and establishing authentication, the server backup program then consults "policy data" which instructs the server backup program as to what sort of a backup operation should occur and which files on the client computer are the subjects of the current backup. It then searches all or a subset of files on the client computer, determining which files should be backed up. For example, a data file which has changed since the backup program was last run may cause that file to be selected for the backup operation. After selecting the files to be backed up, the client backup program transmits those files, using the LAN, to the server backup program. The server backup program then makes an entry in a "backup catalog" for each file received and then stores those files on storage devices attached to the backup server.
The server backup program also carries out several other important operations. One such operation is the maintenance of its storage pools. For example, backup copies of files that were made many months ago may be moved from disk storage to tape storage in order to reduce storage costs. Another important function of the client and server backup programs occurs when the user requests the restoration of a file. The client backup program contacts the server backup program which consults its backup catalog to establish the location of the backup copy of the file. It then returns that file across the network to the client computer which in turn makes it available to the user.
Hardware which is typically needed for implementing a backup system in a client-server system includes: one or more server computers such as PC or workstations and storage mediums such as IBM 3390 magnetic storage system, IBM 3494 tape storage library or IBM 3595 optical library. These libraries which provide automated mechanical mounting and demounting of tape or optical cartridges into read/write drives and retrieve them from or replace them within the storage shelves are sometimes referred to as "jukeboxes".
Despite the recent improvements made in the field of client-server backup systems, several shortcomings have remained in all client-server backup systems including ADSM. One of the shortcomings, as mentioned earlier, is that the very large number of files on the clients now being regularly backed up tend to generate very large amounts of data resulting in large storage requirements and therefore substantially more cost in backing up data. Although systems such as ADSM compress this data on the storage devices, the amount of data remains very large. A second difficulty that is being observed is that the local area network technology is frequently unable to complete transmission of all of the changed files, even in only an incremental backup, to the backup server during the designated period for backup operations (e.g., a night shift). This is due to the bandwidth limitation of the communication network (which might include low speed remote telephony data links) and large amount of data that has to be transmitted from numerous clients to the backup server.
It is apparent now that implementation of an efficient backup subsystem in a computer processing environment is a formidable task and in a client-server environment poses significant challenges of its own. Therefore, there is a need for a new and novel backup method and system in a client-server environment that not only substantially reduces the storage requirement of backup subsystem but also minimizes the burden on the communication link between the clients and the backup server. The present invention addresses these two major deficiencies currently present in all client-server backup subsystem by providing alternative methods and systems which can be used to reduce the amount of data storage required in a client-server backup subsystem and reduces the burden on the bandwidth of the transmission network.